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COST EFFECTIVENESS IN .EVALUATION TECHNICAL ASSISTANCE: 
DIFFERENT ASPECTS OF MEASURING COST AND OUTCOMES 

Gary D. Estes 
Northwest Regional Educational Laboratory 



Background and Problem 

This paper focuses on the Title I (now Chapter I) Evaluation Technical 

Assistance Centers (TACs) to illustrate issues which need to be considered 

prior to projnoting "cost-effective" strategies for innovations or pr5^aros. 

Previous presentations in this symposium examined issues ab^ut^JWCC activities, 

goals and effects including cost-effectiveness. The need for clarifying these 

issues is illustrated through statements from several reports. First Millman 

in Reisner, et al. , (1982) states: 

The cost of TAC assistance is high relative to the amount 
of service provided 

EDs should consider options for reducirfg cost while 
maintaining the current quality of services* 

*t 

Most readers would conclude from the statement that (a) providing services 
was an objective or "effect" expected of TACs, (b) the quality of services was 
satisfactory and (c) the methods used to provide these services were top 
costly for the level provided. Only a vague evaluation of the quality of TAC 
services is implied with a strong reference to cost and level of services. 

Stonehill and Groves (1983) conclude TACs are effective: 

Subtle yet often important influences and revisions on 
program operations have been brought about through the 
implementation of TIERS and through the efforts of TACs. 

They address TAC effects more directly than Millman and without any 

reference to costs. Reisner, et al., (1982) are even more explicit about TAC 



effects: 
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Through technical' assistance centers (TACs), state education 
agencies (SEAs) and local education agencies (LEAs) have 
learned new evaluation approaches and rtew ways to use 
evaluation data to improve educational services. 



This statement explicitly addresses the effectiveness of TACs and again 
does not interpret costs or cost-effectiveness. These three statements 
probably give different impressions of TACs. 

Different effects are implied in the above statements. Stonehill and 
Groves suggest the changes * n programs; Reisner, et al. , focus on learning new 
evaluation approaches and using evaluation data. Millman's evaluation based 
on costs leaves a much less positive impression regarding TACs and implies 
effects are the level of services provided. 

It appears that TACs have some good effects. How good depends on what 
is/is not included as effects and whether or not costs are included in the 
evaluation. Further, TACs have several options for allocating or using their 
resources to produce any combination of desired effects. I want to review TAQ 
costs, effects, and some alternative strategies to highlight the implications 
of different approaches and variables in making decisions about TACs. 

I will: (a) describe alternative cost analysis approaches which might be 
used in conducting a cost analysis, (b) analyze costs for alternative 
technical assistance emphasis, (c) discuss criteria which might be used to 
measure effects, and (d) discuss implications for using cost analyses 
techniques to plan or manage programs so that funding agents, clients and 
program staff are satisfied with the program. 

Levin (1981) provides a manual for conducting cost analyses in evaluating 
educational programs. Table 1, from Levin, describes four approaches in terms 
of distinguishing features, strengths and weaknesses. I will relate them to 
technical assistance centers. A cost effectiveness analysis of technical 
assistance centers might be one which defines effects as the number or 
proportion of districts which submit annual evaluations with sufficient 
quality to be aggregated at state and federal levels, and ^hich compares these 
effects to the costs of services to assist districts to produce these 
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Table 1 
Modes of Cost Analysis* 



Type of 
Analysis 



Distinguishing 
Feature 



Strength 



Weaknesses 



Cost-Effectiveness 
(CE) 



Cost-Benefit 
(CB) 



Cost Utility 
(CU) 



Cost-Feasibility 
(CF) 



Outcomes are measured 
in units of effects. 



All outcomes are in 
monetary values. 



Outcomes are 
measured according to 
subjective judgements. 



Es t ima te whe ther the 
cost of an alternative 
is possible within 
the financial 
constraint. 



The outcomes can be 
psychological or 
physical changes and 
do not require con- 
version to monetary 
values. Analysis 
replicable. 



The unit of effectr 
iveness must be the 
same among alterna- 
tives which address 
a particular goal. 



Single metric allows 
comparison of 
different alterna- 
tives; or comparison 
of projects in 
different areas 
compe t i ng for the same 
resources. The results 
can be expressed in 
internal rate of return, 
net benefits, or cost- 
benefit ratios. 
Analysis replicable. 



Often difficult to 
get outcomes in 
monetary values. 



Can integrate mul- 
tiple outcomes into 
a single value. 



Indicates if further 
consideration of that 
alternative is 
feasible. 



Measures are highly 
subjective; and 
analyses are not 
replicable* 

Does not deal with 
the outcomes of an 
alternative at all. 



a Froip: Levin, H. and Seidman, W«, 1981 
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evaluations. The technical assistance approach which minimized the cost per 

valid/useful report would be chosen as cost-effective. A cost benefit 

analysis would differ in that the effects would be specified in monetary terms 

to determine whether the outcowes in dollars was equal to or greater than the 

cost necessary to obtain the benefits. A technical assistance strategy would 

not only nee<^ to have an optimum tatio, but should also have benefits which 

exceed costs. A' cost utility anfalysis might use effectiveness criteria that 

are converted to a common scale of utility and then relate these criteria to 

the cost for providing technical assistance. Examples of criteria are client 

ratings of satisfaction with TAC services or, subjective assessments of how 

likely the strategy is to produce good reports. The strategy which maximized 

the utility/cost ratio would be desired. A cost feasibility analyses could* 

simply be whether the desired technical assistance services can be provided 

within the available) budget . If yes, then provide or fund the services. 

A further illustration of the differences among the four alternatives is 

obtained by assuming tfiat the cost or the resources available for technical 

assistance are a fixed quantity. Then decisions among alternatives would rest 

on effects. Specifically, the coat effectiveness study mentioned above would 

reveal which approach or alternative produced the greatest number of 

aggregatable district reports. The results from the cost benefit and cost 

utility analyses would be similar except that the criteria for cost benefit 

4 

would be the approach that maximized the outcome in dollar values and the 

criteria for cost utility would be the approach which maximized utility. 

Finally, decision alternatives within the cost feasibility would be to 

determine if the desired technical assistance could be provided with the 

available resources without specif ically . considering effects. 
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It should be clear ^ that the analyses with the exception of cost 

feasibility assume: 

a. it is possible to specify and a^t^e upon program outcomes or 
objectives and to obtain adequate/Or comprehensive measures of these 
outcomes; 44 

b. it is possible to specify costs for the inputs necessary to produce 
the desired outcomes; and 

c. that a relationship between inputs and outcomes exists such that 
differential inputs can be related to predictable outcomes. 

I willv not ^address the issue that resources for technical assistance 
centers raigW be used for other purposes such as reducing the national 
deficit. It is also* not reasonable to express TAC effects in dollars. 
Finally, given the fact that TACs are funded and operational, I will not 
address the cost feasibility or utility of TACs as a system. I will focus on 
cost and effect implications for alternative emphases of TAC services. 

Identifying and Analyzing Costs 

Levin (1981) outlines three questions helpful in identifying costs for 

alternatives: 

1. what are all the resources or ingredients required in order to have a 
program ?.. 

2 . What does it take to have a program ? 

3. What are the program's social cost ? 

He also relates 1:hat ingredients or costs can be distributed across groups 
such as sponsors, service providers and clients. This paper will be limited 
in the following ways. First, costs to the sponsor, i.e., United States 
Education Department (USED) , will be primarily restricted to funds for the 
TACs. Thus, the cost to USED for monitoring the TACs, writing RFPs and 
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selecting the contractors are not addressed. Second, cost to clients will not 
be estimated in monetary terms but will be proportioned among 10 points, 
solely to emphasize differential client costs for the alternatives. Third, 
technical assistance costs will be reported as percents for an unspecified, 
but fixed dollar amount. Alternative approaches to providing technical 
assistance will simply be defined by how this fixed cost is proportioned 
across the cost categories. 

Evaluating Costs . Levin (198L) provides five techniques for evaluating 
cost. These are summarized in Table 2. They are presented here to highlight 
the techniques available and provide brief comments on potential applicability 
to the TAC context. The annualized cost and present value appear to have the 
least usefulness for our purposes or assumptions. An exception might be in 
the field office option in which the initial cost in setting up a field office 
might be annualized across years. A joint cost approach might be useful if a 
technical assistance center was to base its budget or costs on the shared uses 
or business potential from TAC activities. TAC services or costs might be 
provided "below cost" if additional contracts for service, materials, or other 
returns for TAC effort could be projected. This would certainly require a 
major policy change toward federal support for TAC services. 

Market prices and shadow prices are interesting in the context of TAC 
services. Specifically, in trying to value TAC services one might aek, "What 
should they cost?" Millman implied market price , by stating TAC services cost 
more than university professors or consultants working out of their homes. A 
report of CCSSO Consolidation Evaluation Task Force (1982) appears to apply a 
shadow price for TACs through their support for continuing TACs. It is 
important to note, however, that their willingness to support was under the 
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Table 2 

J Valuation Techniques* 



Technique Definition ' Conditions for Use 



Market Prices 


The price tor an ingieaienL 
in the open market* 


Pnr markpt* aoodfl 


Shadow Price 


The value of the sacrificed 
alternatives as indicated by: 

a) tl>e price of an 

£ approximate market or other 
equivalent for a non-market 
good 


For non-market goods 


r 


b) people's willingness to pay 
for an ingredient 




Joint ^osts 


The proportion of an ingredient 1 ^ 
value allocated to the 
alternative under consideration 


For ingredients that 
are put to muiLi^ic 
uses 


Annualized Costs 


An equal payment made annually 
that accounts for multi-year 
projects and foregone interest 


For ingredients with 
'life spans of more 
than one year 




Given an initial cost, C, for an 
item of capital equipment, its 
lifetime, n, and the social rate 
of discount, r, the annualized cost 
of capital is given by a(r, n) 
multiplied by C, where a(r,n) is 




Present Value 


A single figure for a stream of 
future costs discounted at the ' 
appropriate interest rate 


For ingredients that 
have on-going costs 



a Prom Levin, H. and Seidman, W., 1981 
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condition that' the funds were not directly from their resources, and ^AC 
funds" were not available for states' use. In fact, "willingness to pay" or 
shadow price would inevitably be different if clients were given the resources 
to do with as they wanted. 

Costs for Technical Assistance Alternatives 

Following is a list of seven categories or ingredients which are used to 

define the cost for providing technical assistance. 

Personnel ; Personnel costs are the salary and benefits associated with 
supporting staff . 

Travel : Travel costs are those associated with transportation and 
subsistence. 

Phone : Charges for long distance, WATs and other phone expenses. 

Materials : These include materials produced for workshops and 
consultations. It includes costs for duplication and does not include 
labor costs associated with producing or duplicating materials. 

Facilities : These are the office space charges and are proportional to 
the personnel costs. 

Indirect/Fees : General corporate overhead for personnel, management and 
other indirect costs and fees. 

Other : These are costs associated with items such as paying client travel 
for attendence at regional coordinating council meetings and use of 
outside consultants. 

Table 3 provides the proportion of the 1982-83 Region 4 TAC budget 
allocated for each of these categories. The existing or present focus for 
TACs allocates approximately 51 percent of the expenditures for personnel, 12 
percent for travel, etc. Millman (1982) reported the following percentages 
for the 10 Title I TACs from 10-79 through 9-81: 

Salary: 45% 

Travel: 8% , 
Indirect fees: 41% 

r 

OthGr direct: 6% 
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Table 3 
Costa for .TAC Alternatives* 



Cost Existing/ More Emphasis on Field Broker-Purchase 

Category Present Focus Workshops Consultations Letters/Phone Office by Clients 

Personnel 51% - 7% - 4% + 3% +3% +3% 



Travel 



12% +6% +6% - 4% - 4% - 3% 



€ 

Phope 1% + 11 

Materials 3% > 2% - 1 % ♦ 1* 

Facilities 6% - 1% - 1% 



Indirect Fees 23% 



Other 4% 



*Table illustrates how a fixed amount of funds would be distributed differently for 
each alternative. Percents under "Existing" are percents of the 1982-83 TAC budget allocated 
to each category. Percents in columns for the five alternatives are the shift in the current, 
e.g., - 7% for workshops in personnel means 7% less of the total budget would be allocated 
to personnel or that only 44% (51 - 7) would be devoted to personnel. 
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Thus, costs in Table 3 are slightly higher than Millman'a in personnel and 
travel and substantially less in indirect fees* The major discrepancy in 
indirect/fees becomes less if facilities are included as indirect expenses. 
In Table 3 the cost for alternatives to the "Existing Focus" are given as 
percentage shifts across^ the cost categories. The emphasis on workshops, 
increased travel r and materials is six and two percent of the total 
expenditures, respectively. These increases reduce funds by seven percent in 
personnel and one percent in facilities. Thus, a workshop emphasis would have 
fewer staff spending more time providing workshops on-site using more 
materials. The exact figures are not important, e.g., travel could have been 
increased by four or 12 percent. The shifts within categories and how 
resources will be used is the point I hope to make. 

A consultation focus would be similar to workshops in that fewer staff 
would spend a greater proportion of resources in travel. However, 
consultations require fewer materials than workshops. The example in Table 3 
indicates the loss in personnel would f be four percent for consultations versus 
seven percent in workshops. For a staff of 15.0 FTE the workshop option would 
result in a loss of approximately 14 percent of the FTE (.07 f .51 - .14) or 
2.1 FTE . The consultation option would result in a loos of approximately 1.2 
FTE. 

The letter/phone option decreases the face-to-face contact of TACs with 
clients and increases phone or mail contacts. The Table 3 example, reduces 
travel by one-third of the existing costs or four percent of the total budget 
and redistributes this to personnel and phone. Thus, the savings in travel 
could allow more staff to spend greater time (double in the Table 3 example) 
in phone or mail consultations with clients. The additional three percent in 
personnel would result in approximately .9 FTE additional staff over the 
existing option (.03 r .51 x 15.0 FTE - .9 FTE). 



Establishing a field office is ah alternative to maintain high levels of 
face-to-face contact and to reduce travel costs through eloper proximity of 
^ T^ ^to clients^ The example redistributed one-third of the travel costs or 
four percent of the total budget "to increased staff and materials. It should 
be noted that other costs, e.g., facilities* «Jre estimated to remain constant 

under this option. This obviously would depend on the specifics for a field 

K " 
office arrangement. Given these assumptions, the field office would use 

decreased travel costs to support more staff spending more time with clients 

in f ace-to-f^ce consultations. * 

Millman (1982) offered an alternative of having clients broker for TAC 

services to illustrate drastical]tf changing the TAC structure. Briefly, 

clients would have the option of negotiating with a list of approved service 

providers, might use RFPs and the proposal process for major pieces of work, 

and would pay a nominal fee for services requested. This option provides 

another comparison of how costs/resouroes might be effected^ Table 3 assumes 

that more personnel costs would be associated with negotiating , planning 

services and responding to proposals, with consequently fewer resources 

available for travel and less time with clients. 

Table 3 highlights five options to the current technical assistance 

efforts through increased emphases on workshops, consultations, or 

letters/phone or through alternative delivery via a field^ffice or' brokerage 

by clients. Using 15.0 PTE for the current option, the alternatives range 

from 12.9 FTE to 15.9 FTE • Travel costs would range f^pflM^ percent of the 



total budget to eight percent* These figures, i.e., 12.9 j^o 15.9 FTE aSd 8 to 

) 

18 percent for travel, illustrate that the alternatives woiild have 
proportionately greater effect on the type of contact or assistance provided 
than on the number of staff available. That is, the number or amount of 
workshops could be increased by one-half or 50 percent with only a 14 percent 



reduction of 'staff. Finally, these alternatives leaVe little prospect that" 
major savings could be achieved without reductions in personnel. The four . 
percent "savings" in travel for phone/ letter and field office alternatives 
could be used to reduce the overall budget rather than to increase personnel. 



Staff Time * Use of staff time is another way to view costs for technical 
assistance. Since personnel, facilities and indirect fees are directly 
related to personnel costs, it appears that 80 percent of resources used is 
explained by staff and related costs. 

Table 4 summarizes information on distribution of costs for professional 
staff time. Data were taken from the Region 9 TAC Final Report (1983). #ield 
service time is divided into time spent on-site, traveling, and in-house 
preparation for th^ field activity. Other direct technical assistance time is 
categorized by phone/letters and materials development/technical 
investigations. Materials development and technical investigations is the 
time for developing workshops or instructional packages for conducting studies 
for clients. Staff development time primarily includes participating in staff 
meetings and in-house sessions. Admihistration includes activities such as 
performance reviews, supervising budgets and communicating with project 
officers. Other tasks include time for tasks such as attending TAC Directors 
meetings, conducting outreach or awareness activities, serving on inter-TAC 
committees-T^etc. The last three categories might be viewed as indirect 
service or support for the direct service activities. Millman provides 
similar figures for ten TACs in December, 1981 and January, 1982. His 11 
percent for field service probably reflects the lower level for these months 
and differences across TAC. Peak field service periods were September-October 
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and March-May. Again, it should be noted that data in Table 4 are for the 
Region 9 TAC in 1981-82. The alternatives in Table 4 are those described 
earlier for Table 3. 

The percentages for alternatives were based upon the estimates of shifts 
in personnel and travel from Table 3. An example will illustrate the 
derivation of figures in Table 3. The increased travel funds in Table 3 under 
the workshop option will support approximately 50 percent more trips and 
travel, i.e., 50 percent more, time was assumed. I also assumed that 
approximately 50 percent more workshops would be provided with the increased 
on-site time. To calculate the percent of staff time it was necessary to take 
into account the loss of staff under this option. It was estimated that 
approximately .86 of the existing staff would be available to provide the 
increased level of workshops (this figure was obtained by dividing .44 (i.e., 
.51 - .07) by .51 in Table 3. The numerator for onsite activities was 
obtained by adding 10 percent to the 19 percent for 1981-82. The resultant 
29 percent is an estimate of the "absolute time" devoted to onsite workshops 
which would be approximately 50 percent more on-site time than with the 
existing emphases. Dividing thJ 29 percent by .86 results in an estimate that 

34 percent ot the somewhat redufced staff time would be devoted to onsite field 

/ 

service under workshops. This assumed that staff development, administration, 
and "other tasks" total time would be reduced proportionate to the decrease in 
the numbers of staff. Thus, the relative increase in travel arid onsite/tim* 
is offset by reductions in preparation and materials development; workshops 
were assumed' to be available and staff trained/prepared to give them. Tnis 
assumption would certainly be less valid over time due to staff turnover and a 
need to move onto new topics. It is estimated that phone calls and letters 
would remain in approximately the same number but fewer staff would result in 
a slight increase in proportion of time devoted to these areas. In summary, 
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the workshop emphasis would have staff spending greater proportions of their 

time on-site with standard , already developed workshops with les^ nee<3 for 

preparation. One would need to consider the long-term as well as short-term 

efficacy of this approach. 

The consultation option differs from the workshop id proportion of time 

primarily in the preparation and development area^Xft is assumed that 

approximately the same amount of field time would be devoted as under the 

workshop option with the emphases on consultations rather than workshops. { 

This would imply proportionately greater time would be devoted to preparing 

for the individualized consultations and would leave even less time available 
% 

for future materials development or technical investigations. One should 
consider the intensive tailored benefits of consultations to the more general, 
awareness type assistance of workshops for greater numbers of clients. 

/ 

The letters/phone option time distribution was calculated by doubling the 

V 

amount of time devoted to phone calls and letters, reducing the field services 

i 

and maintaining a relatively high level of materials development. The 
rationale for this distribution was that materials "will be much more important f 
in support of phone or letter consultations than when face-to-face assistance* 
is provided. Again, one should begin considering use of materials and phone 
for T.A. in relation to onsite assistance. 

The field office differs from the existing operation primarily in that the 
reduced travel requirements and resources directly contribute to increased 
staff and time available for conducting on-site activities with only marginal 
losses in proportion of time for preparation, phone/letters or development. 
^ Finally, a broker/purchase by clients option shifts personnel time to an 
increased emphases on preparation for field work resulting from planning, 
writing proposals, multiple contracts. m The time presumably would make 
services more responsive or directed to clients' individual needs. Slightly 
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greater administration costs or proportionate time would be incurred due to 
the need^ to respond to RFPs or conduct more negotiations in- deciding on 
specific services. 

Table 4 is also useful in considering the level or amount of services. If 
j&ore services are provided, two questions are raised: (a) Who will provide 
them? and (b) How will service providers use their time? Table 3 data reveal 
only marginal savings can be obtained from non-personnel related categories. 
Table 4 data estimate that even under the mosb intensive option (workshops) 
44 percent of the staff time will be in the field or on-site. If the field 
office option was combined with increased emphasis on workshops, the on-site 
time could be increased more. It is clear, however, that differences among 
options will not effect time with clients nearly as much as whether the 
decision is to spend more time face-to-face with clients and less time 
prep aring, developing, coordinating (i.e., staff development) or a dministering. 

Client Time . Table 4 also summarizes a global look at how client time 

\ 

might be spent in relation to the alternatives. The, figures represent a 
distribution of "ten units of time" across (a) planning/coordinating for 
technical assistance, (b) accompanying or meeting with TAC staff, or 
(c) providing fo^owup to the technical assistance activities. The primary 
variations occur in the broker purchase alternative. Specifically, it was 
estimated that the planning/coordinating demand on clients under this option 
would be approximately twice those for the other alternatives in which clients 
have only one TAC to contact, do not have to write/review RFPs or negotiate 
contracts to secure TAC services. 



15 ^ I i 



4? 



Table 4 

TAC Professional Staff Costs in Percent of Time By Activity 3 





Existing 




More Emphasis on 




Field 


Broker-Purchase 


Activity 


19 81-82 


Workshops 


Consultations 


Letter s/Phone 


Office 


by Clients 


Field Service: 




1 










On-site 4 

il a vc i 

Preparation * 


19% 
6% 
17% 


34% 

i nt 
10% 


34% 

10% 
20% 


11% 
4% 
10% 


24% 
4% 
16% 


11% 
4% 
25% 


Phone fc Letters 


10% 


12% 


12% 


23% 


9% 


10% 


Materials 
Development/ 
Technical 
Investigation 


30% 


16% 


6% 

V 


34% 


28% 


30% 


Staff Development 


6% 


6% 


6% 


6% 


6% 


6% 


Administration 


6% 


6% 


6% 


6% 


7% 


8% 


Other: .TAC 6% 
Directors Meeting, 
Outreach 


fit 


fit 


6% 


61 


fit 


FTC Base 


1.00 


X 

.86 


.92 


1.05 ' 


1.05 


1.05 


Client Time: 














Planning/ 
Coordinating 


3 


3 


3 


3 


2 


6b 


Accompanying 


6 


6 


6 


6 


7 


6 


Follow-up 


1 


1 


1 


1 


1 


1 



a Based on 1981-82 Region 9 Final Report. 

b Assumes more total client time required for this option. 
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The above analyses all assumed a fixed level of resources and varied the 
distribution of the resources. Another approach is to determine whether the 
same services can be purchased with fewer resources. The broker/purchase 
alternative might reduce cost through more competitive offerings of TAC 
services. While possible, this was not analyzed here because (a) there was/is 
a fixed level of resources available for TACs, (b) TACs were funded on a 
competitive basis presumably resulting in the best quality of services for a 
fixed amount, and (c) there was need in this paper to maintain a manageable 
number of variables in highlighting and discussing costs for alternatives. 

Identifying Effects or Program Outcomes 

This paper will draw upon TAC final reports, the external evaluation by 
Millman (1982) and the RFP for TACs/ to highlight commonalities and differences 
in TAC goals or effects. I will use these to stress (ay the importance of 
agreeing on effects at as specific a level as possible and (b) the differences 
in values or evaluations which can occur if effects are left at a general 
level of agreement. 

RFP . The Request for Proposal for Chapter 1 TAC states: 

The TAC emphasis will be on capacity building, e.g., 
assisting the SEAs and LEAs to become more capable of 
designing and carrying out the evaluation of their own 
projects. >. 4 

Although this statement does not contain a specific effects criterion, it has 

direct although somewhat ambigious implications for what the outcome of TAC 

services should be. For example, if the "capacity building" means to 

implement specific evaluation procedures, than an effects criterion might be 

the number of districts or states able to do so. Once accomplished, then 

little need for TACs would exist if one assumes the capacity will maintain 

across years and staff changes. Stonehill and Groves (1983) clearly relate 

that effects for TAC were always intended to be broader than just implementing' 
O 17 
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a specific evaluation. If "capacity building" implies this capacity to 
conduct better, more useful evaluations in general, then other criteria might 
be the extent to which clients change evaluation practices, adopt/adapt new 
strategies, etc. These effects might best be measured through assessing 
movement on a "levels of use" continuum or movement across topics. Murray and 
Quilling (1980) discussed change constructs in relation to assessing technical 
assistance from this perspective and Arter (1983) also discusses changes 
across topics • 

The TAC RFP also provides implicit criteria for effects by describing that 
TACs will provide workshops, onsite consultations, phone consultations and 
related types of services. As will be discussed later, "implicit criteria" 
hece might even better be viewed as "inputs" rather than effects. Finally, 
the RFP states that assistance will cover contents ranging from the required 
evaluation reporting <^stem to more general issues in testing and evaluation. 
Additional points contained within the RFP introduce "cost effective" as a 
criterion: 

...it is essential for the offerers to consider cost 
effective methods of providing services and to describe 
efficient methods of serving the regions. 

This is further clarified by: 

...offerers may want to consider the effectiveness of 
providing field based staff to provide services to SEAs and 
LEAs. Possible mechanisms for providing such staff include 
use of field offices, subcontracts, universities or 
' companies and permanent placement of staff in field 
locations. 

These comments suggest a clear desire to obtain cost effective services. What 
were viewed as effects and what were viewed as the inputs to produce "lofcal 
capacity" was less elear. Specific effects or outcomes were open to 
suggestion by potential contractors and subsequent negotiations or agreements 
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* 

with the USED. Presumably, these negotiations wfculd make explicit the 
relation between goals, effects and inputs intendted to produce specific 
effects. . I 

TAC Perception of Goals and Effects . The NWREI/TAC Region 9 Pinal Report 

summarizes the major goals of TACs to be: 

o assisting SEAs and LEAs to adopt and implement the TIERS, i.e., 

Title I Evaluation and Reporting System 

o encouraging SEAs and LEAs to improve the quality of data used 

in program evaluation* 

o helping SEAs and LEA^ to develop evaluations that are useful 

for local decision making and helping them learn to apply evaluation 

results for that purpose. 

These goals are related to those in the RFP discussed above; again, 

specific criteria for effects are not clear. The final report and 

presentations by Arter (1982 and 1983) rely primarily on effects measured by 

data contained in monthly management reports. These data include number of 

workshops, consultations, clients served, hours on site, and contacts made. 

Arter (1982) summarizes the issues facing TAC in determining impact or effects. 

There are several different approaches that on* can take 

in order to assess TAC impact on evaluation practice. 

One would be to examine case studies of particular 

SEAs and LEAs to determine changes due to TAC assistance. 

Another approach would be to survey the SEAs and 

LEAs and have them self report on the impact the TACs 

have had on their own practices... Por this study I attempt to use 

hard data to assess impact. Specifically, I utilize information 

contained on the contact log with clients available since the 

beginning of the contract. 

Thus, Arter, consistent with the NWREL Pinal Report, relied primarily on the 

quantatitive data readily available from monthly management reports. As with 

the discussion on capacity building, beginning to identify and formalize 

criteria can guide the direction of TAC services. Specifically, a strategy 
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which will maximize the number of workshops, may not be the same strategy 
which will maximize successful implementation of evaluation models and improve 
quality of data and develop useful evaluations. One can argue that the 
workshops, consultations, etc., are the means by which these effects are 
achieved. Maintaining some level of workshops, contacts, etc., might be a 
desired effect if one assumes a direct relation to quality of data, usefulness 
of evaluations, etc. Alternately, effects represented by the level of service 
figures might be the ultimate criteria if the goal were only to provide 
services. Finally, numbers as levels of service might be viewed as the inputs 
necessary with the change in topics Arter (1982) discusses as the effects. On 
the other hand, one might not! 

It is helpful to contrast the differences and difficulty in weighing 
effects if number of clients is the criterion. One client or state might 
**sire TAC assistance with developing materials which the state staff can 
use. The cost/client for this assistance will be high. If TAC were to 
provide workshops using ^e materials rather than develop another set of 
materials, the cost/client would be lower. If effects are defined to be 
changes in evaluation use, either strategy might be cost-effective. A 
cost/client criterion implies the development option is not cost effective. 
Again, it is important to decide what effects are desired. 

Panel Review of TAC Effects . Millman in Reisner, et al., (1982) discusses 
effects of TAC services. 

The goals of TACs are undergoing a large change. At the 
time of the 197 8-79 Performance Review, the TAC objectives 
were to bring about an awareness of the requisite 
evaluation models and their intent, to arrange service 
agreements, to build confidence and capacity among the 
clients, and to facilitate implemenation o| the models. 
More recently, the TACs have shifted their emphasis to 
improving the quality of the evaluation data and making 
these data more usable. 
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Millroan's findings seem to parallel the intents in USED's RFP and TAC 
reports. Thus, consistent with the above discussions, the focus appears to be 
on implementing TIERS , improving quality of data and making evaluations more 
useful. This statement highlights the fact that goals or effect criteria do 
change over time. In this case, the goal of getting clients to agree to 
receive service moved from high priority to an assumed condition with other 
goals or effects taking its place. Care should be taken to not lose sight of 
prerequisite goals or effects accomplished or remaining to be accomplished in 
deciding on effect priorities. 

Millman provides one of the more interesting and critical appraisals of 
TAC effects. Specifically, in a section entitled "Cost and Effectiveness of 
the TACs." Millman presents the data in Table 5. It is possible that these 
data were primarily interred to represent the "costs" in the "cost- 
effectiveness paradigm." Another view, however, is that his relating cogts to 
field visits, hours of assistance, clients served, etc., is the effect part of 
the cost effectiveness ratio and implies that TACs which produced lower cost 

effectiveness ratios will be more desirable. One would tend to view a TAC 

< 

which has $1,090 cost per field unit to be more cost-effective than one with 
$5,180 per field unit. 

It is clear that MilliAan did not intend TAC effects to be judged solely on 
the merit of number of workshops, consultations, etc. He states, "The typical 
SEA evaluator is unquestionably satisified by the TAC performance." He also 
provides a list of areas in which TACs have provided assistance. 

Sunwary . The above discussion suggests there is some agreement among 
sponsors, i.e., USED, TACs and clients about TAC goals. General comments from 
Clients as well as USED suggests TACs have been effective in achieving the 
three goals outlined. Millman (1982) and Arter (1982) discuss effects of 
TACs. Criteria such as level of service provided, e.g., number of workshops, 
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Table 5 
Cost of Client Services a 









(November 


1979 - September 

r 


1981) 




TAC 


Each 

Field 

Visit 


Each Hour of 
Assistance in 


Providing 
the Field 


Each Client 
Secved in 
the Field . 


Each Client 
Hour of 
Service in 
the Field (b) 


Each 
Professional 
Staff Hour 


" — 


$1090 


$230 




$130 


$36 


65 




1270 


310 




180 


58 


37 




2000 


320 




90 


32 


52 




2660 


470 




180 


56 


, 54 


.- 


2680 


450 




110 


41 


30 






J / u 




190 


51 


51 





2950 


560 


• 


200 


49 


45 




2950 


850 




250 


85 


61 




4900 


770 




250 


91 


46 




5180 


900 




* 190 


75 


51 


All 
TACs 

<d) 


$2430^ 


$4 80 




$170' 


$55 


$48 



a From: Millffian "iW82) . 

/ 

^Estimated using a 3-predictor regression model to extrapolate from recently 
acquired client service hour data to the November 1979 - September 1981 cost and 
service data base. 

c Equal to the total expenditure for the TAC divided by the number of professional 
staff hours charged to the project. ^ 

^Weighted average. 
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and time with clients are the roost frequent data discussed. In evaluating or 
making discussions about TACs, the "hard data" appear to gain much more 
visibility or stature than the narrative reports on TAC effects. This is 
unfortunate to the extent that the hard data provide at best an indirect and 
at most a misleading picture of effects in achieving goals. 

Cost and Effect Relations . I have identified criteria which might be used 
to assess the effectiveness of TACs. (Numbers) are from the Region 9 TAC 
1981-82 Final Report: 

a. number of -workshops/consultations (301) 

b. number of phone calls or mail contacts (1,139), 

c. number'of clients served (4,272) 

d. number of contact hours (1,302) 

e. number of reports turned in to the states b\ districts 

f. number of districts included in the state report 

g. number of instances in which services result in 
which a program is modified or in an evaluation ^pproach 

h. client satisfaction with TAC services 

i. amount of local capacity built 

j. spin-offs such as better tests, testing pr 
using materials or ideas 

The first criteria (a-d) have extensive existing data "StB-ltfentioned. The 

number of teports (e) and number of districts (f) are quantifiable and could 

be collected. It is also possible that number of instances (g) or client 

satisfaction (h) could be quantitified allbeit with greater effort and 

possibly less reliability. Other criteria might also be scaled and 

collected. Given a fixed cost for providing services during 1981-82, one 

would arrive at significantly different cost-effects ratios. For example, in 

Table 5 the cost per client served is less than one-tenth the cost per field 

visit. 23 ,i 




The data in Table 5 further illustrates the range of coat-effective 
ratios. How one interpretes the ratios is unclear. Millman (1982) concluded 
that the costs were too high. As stated earlier, it is not clear what 
criteria w«re used in making this judgment. The cost of a technical 
assistance program in the medical fiedd was referenced with additional 
comments about consultant fees- It is likely that the initia^ impression that 
$2,430/field visit, $480/hour of providing assistance in the field were too 
high were only reinforced or supported by these references rather than the 
references forming as the basis for the evaluation- In fact, I "feel" the 
costs are too high as I report them here! 

An evaluation of costs might better benefit from considering some basic 
value assessments rather than using the "semi-shadow" techniques: (a) Are TAC 
staff overpaid?, (b) Do staff use their time appropriately or on 'areas 
desired?, (c) If no, what should be the shift in the use of time (see Table 
3)?, (d) Are indirect costs/fees excessive? and (e) If yes, why were contract* 
given to those contractors? Others could easily expand on the list. I 
propose these questions will likely result in more Constructive implications 
than comparisons of TACs to university profeesofs, in-home consultants, or 
other optionl* not directly related to "what to do about TACS?" 
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Costa and Effects in Relation to Alternatives 

Tables 3 and 4 summarize costs for alternatives to the current TAC 
system. It might be helpful to briefly relate costs to potential effects for 
each (again assuming total costs are fixed and are only redistributed) . These 
effects are summarized in Table 6. A summary of my interpretations follows. * 
It ia inevitable others would roafce different interpretations based on their 
assumptions or values. That fact is a primary point of this pa per— "a need 
exists" to be explicit about desired outcomes, costs for providing services 
and the relation between costs and outcomes . 

Workshops : best way to get large number of clients; efficiency of 

workshops to effect change is either low or unknown 

Consultations; good way to effect change and maintain high level of 

\ 

face-to-face contact; number of clients will be low 
Phone/letters : loss of workshops decreases numbers and face-to-face time; 

if time spent on letters or phone is counted then contact 
time might be high/ effectiveness of letters and phone with 
supporting materials to facilitate use is unknown 
Field office ; lots of pluses implies strong option; potential Retractors 
not raised in table include increased administrative costs, 
splitting resources between field and home office, 
efficient access of field staff to home office, producing 
or adapting materials, and interchange of ideas among staff 
Broker ; negative ratings are based primarily on an assumption that 

the potential for better match of services will not effect 
.... increased cost for securing and coordinating services. 
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Table 6 

Effects of Alternative Approaches on Cost-Effective Measures 



Effect 



Work shot 



Consultations 



Phone/Letters Field Office Broker 



Number of Workshops 
Number of Consultations 



r 



Number of Phone Call 3/ 
Letters 

Number of Clients 
Served 

Number of 'Contact Hcjurs 

Number of Reports 
Completed 

Number of Districts 
With Good Data 

Number of Instances 
Evaluation is Used 

Client Satisfaction 



+ 
0 
0 

++ 

+ 
? 



0 
+ 
0 



+ 
? 



0- 



? 
? 



+ 
+ 
0 



+ 

? 



-/? 



+ ■ Increase in number or level of effect* 
- ■ Decrease in number or level of effect* 
0 ■ No change in number or level of effect, 
? ■ Change not known or predicted* 
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Another use of Table 6 is obtained if one views workshops, consultations, 
phone calls, etc., as the inputs for producing effects which were measured by 
number of reports, client satisfaction, etc. An example will illustrate the 
point. The goal is to get clients to attempt a process evaluation using a 
handbook. It is determined that 300 clients can be served in ten workshops on 
how to use the handbook for the same cost as providing 3,000 clients with the 
handbook via mail. The number of clients attempting a process evaluation 
might be measured through follow-up surveys to the workshop and materials 
mailing. Assume this resulted in estimates that " 10 percent of the clients 
participating in workshops attempted a process evaluation. If at least one 
percent of the clients who received materials attempted a process evaluation, 
then the mail-materials approach was the more cost effective strategy, (In 
the absence of actual follow-up data, literature on change processes or 
relative effectiveness of workshops, consultations or materials as assistance 
strategies might provide estimates of the number or percent who would attempt 
a process evaluation from workshops or materials only.) This example 
highlights the need for information on (a) the costs for delivering different 
types of services and (b) estimates of the effectiveness for the different 
strategies. It would be fairly easy to estimate costs for providing 
workshops, consultations and mailing materials. One could use these to see 
what success rates would be required for each approach to be equivalent. 
These would be better information than simply deciding to serve X number of 
clients or provide Y number of workshops or reduce the cost/workshop by, or to 
Z dollars. 

It is important to reiterate that any decisions will not rest on one 
criteria. In the example above, assume that consultations were the most 
cost-effective approach. If the number of clients served either did not meet 
the sponsors' need to demonstrate the "spread- of services or the client 



support for services was diminished because too few received services, it is 
likely that other approaches would be adopted or included. If so, it would be 
easier to point with confidence that the explicit effect or criteria was 
"spread of services," and was not the number 6f process evaluations conducted 
or was a combination. 

V 

I 

Conclusions and Summary 

One should examine where resources are allocated or used, specify the 
desired outcomes, and hypothesize a relationship between the inputs and 
outcomes prior to making decisions about what is or can be "cost effective" 

y" " ' ■ 

programs. Otherwise, stating that program(s) must become more cost effective, 
is premature, misleading, or counter-productive. In summary, "cost-effective" 
used loosely can be in-effective because decisions or strategies can be 
adopted which will not prbduce the outcomes desired. For example, is the 
effect to: (a) increase percent of time spent with clients, (b) increase 
number of contacts or clients served, (c) increase client satisfaction or 
(d) increase use of evaluation. All might be objectives. It is unlikely that 
equal priority can be given to all. Deciding what inputs will yield the 
desired effect(s) , analyzing the cost for each input and deciding on a balance 
is one approach to make operating "the program needs to be cost effective." 

The exercise of relating cost and use of resorces in Tables 2 and 3 was 
helpful to me in reviewing the relationship between alternative resources and 
uses. Examining the variables which might serve as criteria for effects is 
also enlightening. 

My conclusions are: 

a. It is critical to be informed with data about where and how costs are 
distributed for a program ( s) . 
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b. An explicit agreement on th# effects and criteria for effects is 
necessary if a program is to know how to place priorities among goal? 
and objectives. 

c. Using information from (a) and (b) in making programmatic decisions 
will improve efficiency ^ih accomplishing goals. 

d. Like evaluations, cost-effect studies will be only one piece of 
information which is used in decisions to continue, discontinue or 
modify a program. 

Finally, TAC staff can use cost and effects analyses in identifying or 
selecting strategies which they believe will efficiently or economically lead 
to their goals. The sponsors, USED in this case, can benefit from their own 
applications to communicate desired outcomes to TACs, specify data^ on which 
those outcomes will be evaluated and analyze assumptions implied in adopting a 
technical assistance strategy or policy. Making explicit Outcomes and 
relations of cost to outcomes will increase the probability some goal is 
achieved in which case clients will also benefit. Finally, if all acknowledge 
that this is a dynamic process in which it will never be the case that all 
desired effects can or should be specified a 1 priori, the potential proactive 
or formative use will be maximized and the "well, that's not what we thought 
we were about or you have missed some really important parts in your analysis" 
will be minimized. 
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